Puzzling Out PISA 

What Can International Comparisons Tell Us 
about American Education? 



By William H. Schmidt and 
Nathan A. Burroughs 

I t is no secret that disadvantaged children are more likely to 
struggle in school. For decades now, public policy has focused 
on howto reduce the achievement gap between poorer stu¬ 
dents and more-affluent students. Despite numerous reform 
efforts, these gaps remain virtually unchanged—a fact that is deeply 
frustrating and also a little confusing. It would be reasonable to 
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assume that background inequalities would shrink over the years 
of schooling, but that is not what we find. At age 18, rather, we find 
differences that are roughly the same size as we see at age 6. 

Does this mean that schools can't effectively address inequal¬ 
ity? Certainly not. One of the key factors driving inequality in 
schools is unequal opportunity to learn (OTL) mathematics. In 
previous articles for American Educator and elsewhere, we 
define OTL as the important yet often overlooked relationship 
between mathematics performance and exposure to mathe¬ 
matics content.* 

As we will explain, it is very unlikely that students will learn 
material they are not exposed to, and there is considerable evi¬ 
dence that disadvantaged students are systematically tracked 
into classrooms with weaker mathematics content (e.g., basic 
arithmetic taught in a so-called algebra class). Rather than miti¬ 
gating the effects of poverty, many American schools are exacer¬ 
bating them. 

*See "Equality of Educational Opportunity" in the Winter 2010-2011 issue of 
American Educator and "Springing to Life" in the Spring 2013 issue, both available at 
www.aft.org/ae. 
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Previous work in this area has been limited by the data avail¬ 
able, 1 but the most recent Program for International Student 
Assessment (PISA) study, coordinated by the Organization for 
Economic Cooperation and Development (OECD), opens up new 
opportunities for analysis. The 2012 PISA includes student-level 
measures of mathematics OTL and provides powerful evidence 
of inequality in OTL and its relationship to student performance. 
Specifically, the latest PISA data find that: 

p There is large variation in exposure to mathematics content; 
p OTL is strongly related to student performance; and 
p Lower-income students are generally exposed to less-rigorous 

math. 

It’s not just that lower-income students are less well prepared 
when they enter school; the weakness of their math coursework 
actually keeps them from catching up. 

What is truly fascinating about the PISA results is that this is a 
global phenomenon. In every country, more exposure to formal 
math content was related to better math performance, and almost 
every country showed a statistically significant relationship 
between student socioeconomic background and OTL. In other 
words, the problem we identified in the United States turns out to 
be a problem everywhere. 

One interesting finding of PISA was that most of the variation 
in student performance was within schools rather than between 
them. Here in the United States, we are accustomed to talking 
about "good schools" and "failing schools.” According to PISA, 
this perspective maybe overstated. On average, nearly two-thirds 
of the differences in student achievement in math are found in 
the same school, not in different schools. Much of this difference 
resides between classrooms, as students in the same grade cover 
profoundly different mathematics content—even when their 
classes share the same course title. 2 The United States does stand 
out, but not howyou might expect: here, more like three-quarters 
of the differences in math achievement are within the same 
school. The issue appears to be less unequal schools than unequal 
classrooms. 

These findings should make us reconsider our approach to 
education reform. Educational inequality is not a U.S.-specific 
problem, but some education systems do a much better job than 
we do in coping with the effects of poverty. More important, the 
math content that is taught in the classroom plays a critical role—a 
fact that has received far too little attention and one that we exam¬ 
ine here. 

Dispelling PISA Myths 

PISA is an international assessment that measures 15-year-old 
students’ literacy in mathematics, reading, and science. First 
administered in 2000, PISA is given every three years. The results 
from the most recent assessment, administered to more than 
500,000 students globally in 2012, were released in December 2013. 
Participating governmental entities were the 34 OECD countries, 
including the United States, as well as 28 non-OECD countries (and 
three jurisdictions in China: Hong Kong, Macao, and Shanghai). 
We focus this article on the PISA mathematics results of the 34 
OECD countries. 

The results of the latest PISA study of mathematics were quite 
similar to those of other international assessments: the perfor¬ 


mance of U.S. students (481) was to a statistically significant degree 
below the average of other wealthy OECD countries (494) and 
substantially behind the top-performing countries (such as South 
Korea at 554). Despite several rounds of education reform, the 
standing of the United States is pretty much where it was nearly two 
decades ago. 

The response to these results has been familiar, with advocates 
interpreting them to fit their preconceptions. Some argue that the 
continuing mediocrity of U.S. students in mathematics is a dire 
problem requiring major action (which varies based on the ideo¬ 
logical predisposition of the speaker), while others explain away 


PISA provides powerful evidence 
of inequality in opportunity 
to learn math and its relationship 
to student performance. 


these findings by suggesting that international comparisons are 
unfair because of the greater diversity of American students and/ 
or the greater commitment of the United States to the concept of 
equalizing educational opportunity for all students. 

We can also expect to see that one of the top-performing coun¬ 
tries on PISA will become something of an educational fad, with 
scores of newspaper articles and mounds of policy papers dedi¬ 
cated to understanding the secret of its success—just as previous 
rounds of PISA have witnessed serial infatuations with Japan, 
Singapore, and Finland. This isn’t to say nothing has been learned 
from these countries, of coursed 

While certainly understandable, these reactions all rather miss 
the point. Before digging into what PISA can usefully tell us about 
mathematics learning in the United States and how we might 
improve it, let’s first dispel a few misconceptions. First is the long- 
held belief that the weak-to-middling scores of U.S. mathematics 
students can be explained by a difference in who takes the test. It’s 
amazing how often one hears the assertion that other countries 
only allow their elite students to take PISA while the United States 
ensures that students from all academic levels are tested. 

The reality is that every OECD country participating in PISA or, 
for that matter, TIMSS (the Trends in International Mathematics 
and Science Study, another prominent international assessment) 
must meet very strict requirements in terms of student participation 
in order to be included. The organizers of the study are extremely 

"Tor more on what the United States can learn from Finland, Singapore, and Japan, 
see "A Model Lesson" in the Spring 2012 issue of American Educator, "Beyond 
Singapore's Mathematics Textbooks" in the Winter 2009-2010 issue, and "Growing 
Together" in the Fall 2009 issue, all available at www.aft.org/ae. 
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sensitive to the problem of sample bias. Without getting too techni¬ 
cal, PISA is given to a random sample of all schools in a country, 
and within each sampled school, to a random sample of all 15-year- 
olds. The researchers conducting PISA make sure that the students 
taking the test accurately reflect the whole population of 15-year- 
old students in each country. 

A common misunderstanding of the nature of international 
test scores often results in an exclusive focus on the "horse race" 
results of PISA: ranking the nations by their scores and trying to 
discern which ones are doing well and which are doing poorly. 
Making such comparisons is tempting and reflects our similar inter¬ 
est in comparing how well countries perform in the Olympics and 
the World Cup. But country rankings on PISA are not the same thing 
as comparing win-loss records for sports teams. 

As we discuss below, most of the variation in student perfor¬ 
mance is within countries, not between countries. Yes, affluent 
students in Japan do better than affluent students in Germany, 
but the gap between richer and poorer students within either 


We now have reason to believe 
that tracking is not just a problem 
with American schools but also a 
global problem. 


country is far greater than the gap between countries. As a result, 
comparing cross-country variations rather than rankings based 
on PISA scores might be the most useful of international compari¬ 
sons. Comparing a group of higher-performing countries to others 
on a few key metrics, such as gaps between richer and poorer 
students, to see what general patterns emerge contributes to a 
deeper understanding of key educational issues within each 
country and around the world. 

We should also resist the temptation to assume that the U.S. 
education system has seen no changes in score in the last dozen 
years. Although the U.S. PISA mathematics ranking is essentially 
unchanged, there are signs of progress. For example, on the 2003 
PISA, the performance of U.S. students was to a statistically sig¬ 
nificant degree below the OECD average on all four mathematics 
content subscales: (1) change and relationships, (2) space and 
shape, (3) quantity, and (4) uncertainty and data. 3 Nine years 
later, the mathematics performance of U.S. students was statisti¬ 
cally indistinguishable from the OECD average on two mathe¬ 
matics subscales: change and relationships (which is closely related 
to algebra), and uncertainty and data (which is closely related to 
probability and statistics). This represents notable if unspectacular 
progress. Although we cannot say with certainty, the improved 


U.S. performance in algebra maybe linked to the greater emphasis 
on algebra topics in state eighth-grade curriculum standards 
starting about a decade ago. 

That said, the United States still has a way to go in ensuring all 
students are exposed to algebra in eighth grade. As we have writ¬ 
ten previously, such exposure prepares them for higher levels of 
math in high school and postsecondary education. According to 
our research, algebra and geometry are topics taught in eighth 
grade in virtually all of the countries that participate in TIMSS, 
but in the United States, there is great variability in what math 
content students learn in eighth grade. We have found that by 
international standards, our eighth-grade students are too often 
taught sixth-grade math content. 

The Common Core State Standards in math, however, give us 
hope in that they resemble the standards ofhigh-achieving coun¬ 
tries by exhibiting the key features of coherence, rigor, and focus. 
The emphasis these new standards place on algebra is also 
encouraging. For instance, an operations and algebraic-thinking 



domain for grades kindergarten to 5 lays a foundation for algebra 
in eighth grade. 

A Look at PISA and TIMSS 

As we see it, one important benefit of PISA is that its data can be 
used to draw tentative conclusions about what influences student 
learning for good or ill. PISA shows us that what students are 
taught—the content of mathematics instruction—critically influ¬ 
ences what students know. Just as important, it reveals that educa¬ 
tional opportunities related to the coverage of that content widely 
vary in every country, and that students from disadvantaged back¬ 
grounds are systematically exposed to weaker mathematics con¬ 
tent, worsening educational inequality. 

Readers of our earlier pieces in American Educator might be 
thinking that this all sounds familiar, and it should. In those 
pieces, we wrote about opportunity to learn and about how 
American schools are characterized by pervasive inequality in 
OTL—inequality that is strongly associated with student socio¬ 
economic background. 

Although the explicit tracking of U.S. high school students has 
generally diminished, our studies indicate that it is still a very com¬ 
mon but often overlooked practice. 4 With the most recent PISA 
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results, we now have reason to 
believe that tracking is not just a 
problem with American schools but 
also a global problem. 

The foundation for studying OTL 
internationally is rooted in TIMSS, 
which allowed us to identify the 
strong relationship of OTL to stu¬ 
dent learning more than a dozen 
years ago. 5 But there are limitations 
to how far an analysis could go using 
TIMSS data. In TIMSS, the measure 
of OTL was based on a survey of 
teachers in a small number of ran¬ 
domly sampled classrooms within 
each school. The newest PISA, by 
contrast, asks a random sample of 
all students at a school, and there¬ 
fore from multiple classrooms, 
about the mathematics content to 
which they had been exposed, 
whether formal mathematics, 
applied mathematics, or word 
problems. 

While PISA questions are less 
extensive than the ones asked in 
TIMSS, PISA includes questions 
about a student’s family back¬ 
ground, permitting the develop¬ 
ment of an index of student 
socioeconomic status (the PISA 
educational, social, and cultural 
index) capable of being applied 
across countries. The advantage of 
these questions is that we can now 
study inequalities in OTL and stu¬ 
dent socioeconomic background, 
and the relationship between them, 
in a much more detailed way, one 
that more fully represents the diver¬ 
sity in schooling within countries. 

A further distinction between 
PISA and TIMSS is in how they 
define the idea of opportunity to 
learn. In the original TIMSS (1995), 
OTL was defined as (1) exposure to 
mathematics topics, and (2) the 
amount of time devoted to those 
topics by teachers. In the latest PISA 
study, OTL is identified as familiar¬ 
ity with and exposure to a small set 
of key mathematics topics (much 
like the list of topics found in TIMSS) 
as well as real-world applications 
and word problems. The mathemat¬ 
ics topics are mainly those typically 
found in grades 8 through 12 defin¬ 
ing the academic content of the 


Table 1. 2012 PISA Results in OECD Countries and Select Non-OECD Countries and Regions 


Country 

Mathematics 

Literacy 

Exposure to 
Formal 

Mathematics 

Within- 
Country 
Variation in 
Formal Math 

% OTL 
Variation 

Within-School 
for Formal 

Math 

Exposure to 
Applied 
Mathematics 

Exposure 
to Word 
Problems 

OECD Countries 

Australia 

504 

1.7 

134 

80% 

2 

1.8 

Austria 

506 

1.5 

129 

57 

1.8 

2.1 

Belgium 

515 

1.8 

141 

72 

1.9 

1.9 

Canada 

518 

2 

100 

89 

2.1 

2 

Chile 

423 

1.7 

92 

75 

2.1 

2 

Czech Republic 

499 

1.8 

78 

71 

1.6 

1.6 

Denmark 

500 

1.6 

98 

88 

2 

1.9 

Estonia 

521 

2 

56 

92 

1.8 

1.8 

Finland 

519 

1.7 

96 

88 

1.7 

2.1 

France 

495 

1.9 

87 

* 

2 

2.1 

Germany 

514 

1.7 

118 

67 

2 

2 

Greece 

453 

1.9 

92 

93 

1.9 

1.3 

Hungary 

477 

2 

80 

72 

1.9 

2 

Iceland 

493 

1.1 

105 

96 

2 

2.4 

Ireland 

501 

1.5 

100 

91 

1.9 

1.8 

Israel 

466 

1.8 

111 

80 

1.8 

1.7 

Italy 

485 

1.8 

107 

68 

1.8 

1.7 

Japan 

536 

2.1 

61 

72 

1.7 

1.6 

Luxembourg 

490 

1.4 

138 

86 

1.9 

2 

Mexico 

413 

1.8 

117 

82 

2.2 

1.8 

Netherlands 

523 

1.5 

123 

68 

2.1 

1.6 

New Zealand 

500 

1.5 

139 

83 

2 

1.6 

Norway 

489 

* 

* 

* 

1.8 

1.8 

OECD Average 

494 

1.7 

100 

80 

1.9 

1.9 

Poland 

518 

1.8 

82 

92 

2 

2 

Portugal 

487 

1.7 

100 

90 

2.2 

1.5 

Slovak Republic 

482 

1.7 

86 

67 

1.9 

2 

Slovenia 

501 

1.9 

86 

79 

1.9 

2.1 

South Korea 

554 

2.1 

74 

74 

1.8 

1.7 

Spain 

484 

1.9 

119 

88 

2 

2.2 

Sweden 

478 

0.8 

86 

92 

1.7 

1.9 

Switzerland 

531 

1.4 

137 

60 

1.9 

2.1 

Turkey 

448 

1.9 

83 

85 

2 

1.3 

United Kingdom 

494 

1.6 

118 

82 

1.9 

1.9 

United States 

481 

2 

113 

90 

2 

1.8 

Select Non-OECD Countries and Regions 

Brazil 

391 

1.4 

139 

72% 

2 

1.5 

Chinese Taipei 

560 

2 

89 

82 

1.7 

1.5 

Colombia 

376 

1.7 

138 

84 

2.2 

1.9 

Hong Kong-China 

561 

1.8 

111 

93 

1.8 

1.4 

Indonesia 

375 

1.6 

91 

82 

2.3 

1.9 

Russian Federation 

482 

2.1 

45 

95 

2 

2 

Shanghai-China 

613 

2.3 

57 

83 

1.6 

1.3 

Singapore 

573 

2.2 

113 

83 

2 

1.6 


SOURCE: PISA 2012 DATABASE, HTTP://PISA2012.ACER.EDU.AU. 


^Norway did not provide OTL data and therefore was excluded from the analyses. Data from France do not permit 
within-school analysis. 
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lower- and upper-secondary curriculum. The OECD labels this 
“formal mathematics." 6 While TIMSS assesses formal mathematics 
knowledge (including the concepts, skills, algorithms, and prob¬ 
lem-solving skills typically covered in schools), PISA assesses 
mathematics literacy, which is defined by the OECD as “an indi¬ 
vidual's capacity to formulate, employ, and interpret mathematics in a 
variety of contexts. It includes reasoning mathematically and using 
mathematical concepts, procedures, facts, and tools to describe, 
explain, and predict phenomena. It assists individuals in recognising 
the role that mathematics plays in the world and to make the well- 


Figure 1. The Relationship between Formal Mathematics and 
Mathematics Literacy 

(OECD average = 100) 
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Larger values indicate that exposure to formal mathematics is more strongly 
related to student learning in that country, compared with the OECD average. 
For example, in the United States, the United Kingdom, and South Korea, one 
would find an extremely strong correlation between students' exposure to 
formal math and their mathematics performance, whereas in Sweden, 
Slovenia, and Iceland, more formal math exposure is still correlated with 
higher performance, but the correlation is not as strong. 

SOURCE: ANALYSIS OF PISA 2012 DATABASE, HTTP://PISA2012.ACER.EDU.AU. 


founded judgments and decisions needed by constructive, engaged, 
and reflective citizens." 

The Relationship of OTL to Performance 

First, we used PISA to examine how exposure to formal mathe¬ 
matics, applied mathematics, and word problems relates to 
mathematics literacy (see Table 1 on page 27). A comparison of 
country averages for these three OTL variables reveals consider¬ 
able variation across countries on the emphasis placed on each, 
as measured on a 0 to 3 scale. Among the 33 OECD nations that 
participated in the study of OTL (Norway did not 
collect OTL data, while data from France do not 
permit within-school OTL analysis), lapan and 
South Korea had the highest average for formal 
mathematics (2.1) and Sweden the lowest (0.8). 
Portugal and Mexico averaged 2.2 on applied math¬ 
ematics compared with the Czech Republic's 1.6, 
while Turkey and Greece placed the least emphasis 
on word problems (1.3) and Iceland the most (2.4). 
A comparison across countries suggests that those 
education systems that spent the most time on 
applied mathematics tend to have lower average 
PISA scores—a relationship that is statistically 
significant. 

However, as we mentioned earlier, the ranking 
of countries can be quite misleading. For example, 
a different story emerges when we focus on the pat¬ 
terns within the OECD countries. We found that 
within countries, all three measures of opportunity 
to learn—formal mathematics, applied mathemat¬ 
ics, and word problems—had a statistically signifi¬ 
cant positive relationship to student performance. 7 
In other words, when students had more opportu¬ 
nities to study formal mathematics, applied math¬ 
ematics, and word problems, their performance on 
PISA tended to increase, no matter in which coun¬ 
try that student happened to live. 

Exposure to word problems had a small positive 
association with PISA scores, while formal math¬ 
ematics had a very strong positive relationship, 
with an estimated average effect size that was 
around half a standard deviation. For the United 
States, the relationship of formal mathematics to 
performance is particularly strong—around two- 
thirds of a standard deviation (see Figure 1). In 
short, PISA strongly suggests the importance of 
formal mathematics content. 

| The effect of applied mathematics was more 

1 60 complicated. Applied mathematics was associated 

with higher performance up to a certain point, after 
which additional exposure to applied mathematics 
had a negative relationship. Generally among 
OECD countries, increasing from no exposure to 
moderate exposure was associated with a substan¬ 
tial increase in student performance (approaching 
half a standard deviation), beyond which there are 
limited gains or even drops in performance with 
more frequent exposure. In other words, after a 
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certain point, more work in applying math actually is related to 
lower levels of mathematics literacy. 

The small positive relationship of word problems and the posi¬ 
tive (but more complicated) relationship of applied mathematics 
held for many of the PISA countries. However, the positive rela¬ 
tionship of formal mathematics to student outcomes was far more 
powerful and much more consistent, holding in all education 
systems (OECD and non-OECD countries and regions that par¬ 
ticipated in PISA). 

One reason for the stronger relationship of exposure with for¬ 
mal math might be that students need to be very comfortable with 
a mathematical concept before they can apply it in any meaning¬ 
ful way. For example, to calculate what percentage of one’s income 
is going to pay for housing or childcare, or any other major 
expense, a person must have a clear understanding of how pro¬ 
portions work. It appears that a thorough grounding in formal 
mathematics concepts is a prerequisite both to understanding 
and to using mathematics. 

What all this implies is that while embedding mathematics 
content in word problems or in real-world contexts may improve 
students’ performance, it is the content of the mathematics 
instruction itself that is most crucial. 



Variation in Opportunity to Learn 

Earlier, we noted the great variation in mathematics performance 
within OECD countries. There is also tremendous variation in expo¬ 
sure to formal mathematics content (as shown in Table 1 's “Within- 
Country Variation in Formal Math” column), ranging from Belgium 
(41 percent above the OECD average variation) to Estonia (44 
percent below the OECD average variation). As we explore in more 
detail below, the United States is 13 percent above the OECD aver¬ 
age variation. PISA demonstrates quite convincingly that some 
countries do a much better job than others of making sure all of 
their students have roughly equal access to rigorous mathematics 
content, which includes formal mathematics. 

This brings us to the problem of educational inequality. Educa¬ 
tion has traditionally been viewed as a way of establishing a "level 
playing field” among children from different backgrounds. 8 The 
hope has been that access to good schools will ensure equality of 


opportunity, so that personal merit rather than family income will 
determine the course of one’s life. This vision has played a funda¬ 
mental role in America’s self-understanding. 

However, the results of PISA confirm a growing body of 
research indicating that the U.S. education system is not living up 
to the responsibilities we have placed upon it, not because stu¬ 
dents, parents, and teachers aren’t doing their best, but because 
the education system has not succeeded in ensuring equality of 
educational opportunity. Not only do student scores vary tremen¬ 
dously, but so too does exposure to formal mathematics content. 
Further, this is a problem everywhere, not just in the United States. 
And sadly, although some countries are better at more evenly 
distributing opportunities to learn math, none of them has man¬ 
aged to eliminate these inequities entirely. 

Lower-Income Students 

Are Exposed to Weaker Content 

These educational inequalities are in fact strongly associated with 
student socioeconomic status. Ideally, we would hope that low- 
income students would receive at least equal if not greater educa¬ 
tional opportunities to catch up with their peers. Instead, in every 
PISA-participating country, poorer students received weaker math- 


Some countries do a much better 
job of making sure all of their 
students have roughly equal 
access to rigorous math. 


ematics content. School systems across the globe aren't ameliorat¬ 
ing background inequalities; they’re making them worse. Our 
analysis of PISA data suggests that exposure to formal mathematics 
is at least as important as student background in building student 
mathematics literacy. Theoretically, OTL could be used to mitigate 
the effects of student poverty; instead, we find the opposite. 

The severity of educational inequality varies appreciably across 
countries, whether comparing variations in OTL or the influence 
of student socioeconomic status for different countries. There is 
also a big difference in how education systems, either by design or 
by consequence, contribute to these inequalities. For example, in 
some countries, the inequalities between schools are greater than 
others (Austria has more variation between schools than, say, Ice¬ 
land). There are also substantial gaps in opportunity to learn 
between high- and low-income schools, with the smallest gaps 
among OECD systems in Estonia and the largest gaps in Austria. 
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These inequalities are related to average country performance: 
systems with larger between-school (defined by high versus low 
student socioeconomic status) differences in OTL also have larger 
between-school differences in mathematics literacy. 9 

It is vital to remember that, in every country, most of the varia¬ 
tion in educational opportunity is within schools, not between 
them. On average, about 80 percent of the variation in OTL among 
OECD countries is within schools (see Table 1). The fact that most 
of the inequalities in mathematics content are within schools sug¬ 
gests that attempts to reduce educational inequality that focus on 
high- and low-performing schools will have limited effects. 

A More Detailed Look at the United States 

Although by many metrics the United States is quite similar to other 
countries, there are a few areas in which it does stand out (and not 
for the better). The United States appears to have greater inequality 
in exposure to mathematics content than do other education sys¬ 
tems. It has a 13 percent higher total variation in formal mathemat¬ 
ics OTL than the OECD average—the 12th largest variation among 
the 33 OECD systems. As we might expect, the greater variation in 


In every country, most of the 
variation in educational 
opportunity is within schools, 
not between them. 


United States is characterized by lower between-school inequality 
than other countries. For years, the discussion of educational 
inequality and its association with student poverty has been con¬ 
centrated on the problem of "failing schools," the implication being 
that most of the inequities in the American education system are 
the product of differences between schools. This belief may have 
led some to suppose that the problems in the U.S. educational sys¬ 
tem are isolated, local failures, and not a failure of the U.S. educa¬ 
tion system as a whole. Although it is the case that U.S. students in 
schools with more-disadvantaged students are exposed to weaker 
mathematics content than students in more-affluent schools, this 
is a smaller problem in the United States than it is in other educa¬ 
tional systems. 

However, the PISA data reveal that the between-school inequal¬ 
ity in student performance and student opportunity is dwarfed by 
within-school inequality. Three-quarters (76 percent) of the varia¬ 
tion in student achievement is actually within school (compared 
with an OECD average of 64 percent), and 90 percent of the varia¬ 
tion in opportunity to learn formal mathematics is within school 
(compared with an OECD average of 80 percent). These figures 



OTL among U.S. students is associated with a higher-than-average 
relationship between exposure to formal mathematics and math¬ 
ematical literacy (greater exposure increases math literacy), where 
the United States ranks sixth among OECD countries. Inequalities 
in mathematics instruction therefore play a somewhat larger role 
in accounting for educational inequality in the United States than 
in other nations. 

What is most notable is the counterintuitive finding that the 

Table 2. 2012 PISA Results for Participating U.S. States 


place the United States among the nations with the highest share 
of within-school inequality—seventh among OECD countries for 
OTL inequality and 10th for inequality in student outcomes. 

Another feature of the United States that may differentiate it 
from other countries is the decentralized character of American 
schooling. While other nations have federal systems, the United 
States has long been noted for its extremely fragmented educa¬ 
tional structure. 10 This decentralization of educational structure 
has been accompanied by great variation in edu¬ 
cational standards across states as well as major 
differences in the content of mathematics instruc¬ 
tion across schools even in the same state. 11 

Over the last several decades, however, state 
governments assumed an ever-greater share of the 
responsibility for school finances, administration, 
and curriculum. The United States is not one large 
education system, but (at minimum) 50 autono¬ 
mous ones. We know from the National Assessment 
of Educational Progress (NAEP), conducted by the 


State/Country 

PISA 

Exposure to 
Formal 

Mathematics 

% PISA Variation 

between Schools 

% OTL Variation 

between Schools 

Connecticut 

506 

2.07 

24% 

13% 

Florida 

467 

1.98 

17 

5 

Massachusetts 

514 

2.07 

30 

19 

United States 

481 

2.00 

11 

11 


SOURCE: PISA 2012 DATABASE, HTTP://PISA2012.ACER.EDU.AU. 
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U.S. Department of Education’s National Center for Education 
Statistics, that just as there is considerable variation across coun¬ 
tries, there is also variation among U.S. states. One of the fruits of 
the most recent PISA is that three states—Connecticut, Florida, and 
Massachusetts—agreed to participate in the study with full statisti¬ 
cal samples (rather than being lumped in with students from other 
states). This means we can treat them as independent systems (or 
"countries") for purposes of comparing them to the U.S. average 
and to other countries' education systems (see Table 2 on page 30). 

What we find is that, while some interesting characteristics dis¬ 
tinguish these states from one another and from other systems, they 
share most of the same general patterns we find in the rest of the 
world. Confirming the NAEP results, Massachusetts has higher 
average scores than the rest of the United States (514 vs. 481), 
although it is not among the very highest performers on PISA. 

For all three states, most of the variation in student perfor¬ 
mance and opportunity to learn is within rather than between 
schools, and OTL is to a statistically significant degree related to 
student outcomes, even controlling for student socioeconomic 
background. In each of these three states, the same basic pattern 


The strong relationship between 
opportunity to learn and student 
outcomes suggests that schools 
really do matter. 


emerges that we find in the United States as a whole and in other 
countries: inequalities in formal mathematics OTL exacerbate 
socioeconomic inequalities. In analyzing the differences between 
lower- and higher-income students (defined by the U.S. average), 
we found large gaps in both opportunity to learn and PISA scores, 
whether analyzed between schools, within schools, or a combina¬ 
tion of the two. 

Our analyses indicated that although within-school relation¬ 
ships are still more important, school-level factors play a much 
greater role in Massachusetts than in the other two states. For 
example, 30 percent of the variation in PISA scores and 19 percent 
of the variation in OTL were attributable to school-level factors. 
These findings are only suggestive but do point to some worthwhile 
avenues for exploration. For example, what is it about the Massa¬ 
chusetts educational system that gives schools greater importance? 
And what features of Florida’s educational governance have 
resulted in lower inequality? 

In addition, we should exercise caution when comparing sub¬ 
national units to national ones. At the moment, Shanghai is getting 
a great deal of attention due to its high PISA scores—but Shanghai 


is not the whole of China any more than Massachusetts represents 
the entire United States. Nor does it necessarily follow that states or 
cities have achieved their status because of educational practices 
or policies. For example, the close PISA scores of the Italian Lom¬ 
bardy region (517) and Massachusetts (514) may pardy reflect their 
demographic similarities, in particular their relative wealth. 

C areful analyses of PISA data can tell us a great deal more 
than which country is currently at the top of the inter¬ 
national standings. Research based on PISA presents 
strong evidence that the United States systematically 
disadvantages lower-income students by depriving them of strong 
mathematics content, but it also tells us that this is a global phe¬ 
nomenon. In most respects, the United States is not that different 
from other countries. 

PISA also includes some real surprises that should prompt us 
to re-examine our approach to education reform. Although it 
confirms the great importance of the content of instruction, PISA 
cautions us that with respect to the inclusion of real-world appli¬ 
cations, more is not necessarily better. To that end, we must not 
overly concentrate on such applications at the expense of teaching 
mathematical content. It also calls into question the idea that 
tracking has decreased in American schools. 

PISA does provide reason for optimism, however. The strong 
relationship between opportunity to learn and student outcomes 
suggests that schools really do matter. Some education systems 
are much more effective in minimizing educational inequality, a 
fact which, in the United States, should inspire admiration as well 
as a renewed commitment to the challenge of education reform 
in the service of quality and equality. □ 

Endnotes 

1. The Trends in International Math and Science Study (TIMSS) surveys student knowledge and 
exposure to mathematics content at the country, school, and classroom levels, but does not 
provide student-level information. 

2. William H. Schmidt, Curtis C. McKnight, Richard T. Houang, HsingChi Wang, David E. Wiley, 
Leland S. Cogan, and Richard G. Wolfe, Why Schools Matter: A Cross-National Comparison of 
Curriculum and Learning (San Francisco: Jossey-Bass, 2001). 

3. A description of the four content subscales can be found in Organization for Economic 
Cooperation and Development, PISA 2012 Results, vol. 1, What Students Know and Can Do: 
Student Performance in Mathematics, Reading and Science (Paris: OECD, 2014). Change and 
relationships: This category emphasizes "relationships among objects, and the mathematical 
processes associated with changes in those relationships" (95). Space and shape: This category 
emphasizes "spatial relationships among objects, and measurement and other geometric 
aspects of the spatial world" (103). Quantity: This category emphasizes "comparisons and 
calculations based on quantitative relationships and numeric properties of objects and 
phenomena" (105). Uncertainty and data: This category emphasizes "interpreting and working 
with data and with different data presentation forms, and problems involving probabilistic 
reasoning" (105). 

4. William Schmidt and Curtis McKnight, Inequality for All: The Challenge of Unequal 
Opportunity in American Schools (New York: Teachers College Press, 2012). 

5. Schmidt et al.. Why Schools Matter. 

6. From William H. Schmidt, Pablo Zoido, and Leland Cogan, "Schooling Matters: Opportunity 
to Learn in PISA 2012," OECD Education Working Papers, no. 95 (Paris: OECD, 2013), 7: "Three 
indices were developed from the student questionnaire items: formal mathematics, an indicator 
of the extent of student exposure to algebra and geometry topics in classroom instruction; word 
problems, an indicator of the frequency that students encountered word problems in their 
mathematics schooling; and applied mathematics, an indicator of the frequency students in their 
mathematics classes encountered problems that required the application of mathematics either 
in a mathematical situation or in an every day, real world context." 

7. The within-country analyses were done with a hierarchical model at the between- and the 
within-school levels. The following types of relationships were found at both levels, but here we 
report on the within-school results where most of the variation in OTL and performance was 
present. When referring to the "average," this relates to the OECD countries only. 

8. Schmidt and McKnight, Inequality for All. 

9. Schmidt, Zoido, and Cogan, "Schooling Matters." 

10. William H. Schmidt, Curtis C. McKnight, and Senta A. Raizen, A Splintered Vision: An 
Investigation of U.S. Science and Mathematics Education (Dordrecht, Netherlands: Kluwer 
Academic Publishers, 1997). 

11. Schmidt and McKnight, Inequality for All. 


AMERICAN EDUCATOR | SPRING 2015 31 



